[Scalability] Enable Horizontal Pod Autoscaler (HPA) for Alcor deployment #679
Conversation
| metadata: | ||
| labels: | ||
| k8s-app: metrics-server | ||
| name: metrics-server | ||
| namespace: kube-system | ||
| --- |
There was a problem hiding this comment.
@kevin-zhonghao Kevin, where are those metrics stored? Is it possible to visualize those metrics somethere?
There was a problem hiding this comment.
@xieus Metrics Server collects resource metrics from Kubelets and exposes them in Kubernetes apiserver through Metrics API for use by Horizontal Pod Autoscaler. It is not used to store any data, it is more likely an API to get current resource usage status
There was a problem hiding this comment.
@kevin-zhonghao @xieus Is that similar to Elastic's Meatricbeat? https://www.elastic.co/blog/kubernetes-observability-tutorial-k8s-metrics-collection-and-analysis
There was a problem hiding this comment.
@cj-chung A little similar. In general, we don't use metrics-server as monitoring solution or as a source of monitoring solution metrics. Currently it is just used by HPA.
| policies: | ||
| - type: Percent | ||
| value: 100 | ||
| periodSeconds: 15 |
There was a problem hiding this comment.
Does periodSeconds mean the time interval to check the percentage number?
There was a problem hiding this comment.
Does periodSeconds mean the time interval to check the percentage number?
Not really, periodSeconds: 15 above means it can reduce pods by up to 100% in 15 seconds.
| # The autoscaler will choose the strategy that affects the minimum number of Pods | ||
| selectPolicy: Min | ||
| scaleUp: | ||
| stabilizationWindowSeconds: 0 |
There was a problem hiding this comment.
The value of stabilizationWindowSeconds differs in the ScaleUp and ScaleDown policies. Is it the best practice to set stabilizationWindowSeconds = 0? Does 0 meaning that the autoscaler will always respond to changes immediately?
There was a problem hiding this comment.
The stabilization window is used to restrict the flapping of replicas when the metrics used for scaling keep fluctuating.
When the metrics indicate that the target should be scaled down the algorithm looks into previously computed desired states and uses the highest value from the specified interval.
For example,
here we set up
'scaleup:
stabilizationWindowSeconds: 0'
It should scale up the pods immediately if need.
and we set up
'scaledown:
stabilizationWindowSeconds: 300'
When current metrics indicate that we could scale down the pods, HPA will consider the state of past within 300 seconds to determine if we can scale down now.
[Data Plane Mgr] Fix issues from test scenario 4.5 (futurewei-cloud#688)
|
Some preliminary test in the Medina cluster appears to be quite promising. The current 0.19 release already contains a lot of new features therefore we decided to move this feature to 12/30 release. |
|
@yanmo96 We need to test this PR and get it merged by 2/28. |
Configure cluster and network configuration to meet the requirements on using Metrics Server:
Deploy Metrics-Server in cluster.
Deploy Horizontal Pod Autoscaler for each deployment with customized hpa yaml.